<!DOCTYPE html>
<html lang="en-us">
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    
<meta charset="UTF-8">
<title>Dynamically Updatable Indices | Elasticsearch: The Definitive Guide [2.x] | Elastic</title>
<link rel="home" href="index.html" title="Elasticsearch: The Definitive Guide [2.x]">
<link rel="up" href="inside-a-shard.html" title="Inside a Shard">
<link rel="prev" href="making-text-searchable.html" title="Making Text Searchable">
<link rel="next" href="near-real-time.html" title="Near Real-Time Search">
<meta name="DC.type" content="Learn/Docs/Legacy/Elasticsearch/Definitive Guide/2.x">
<meta name="DC.subject" content="Elasticsearch">
<meta name="DC.identifier" content="2.x">
<meta name="robots" content="noindex,nofollow">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <script src="https://cdn.optimizely.com/js/18132920325.js"></script>
    <link rel="apple-touch-icon" sizes="57x57" href="/apple-icon-57x57.png">
    <link rel="apple-touch-icon" sizes="60x60" href="/apple-icon-60x60.png">
    <link rel="apple-touch-icon" sizes="72x72" href="/apple-icon-72x72.png">
    <link rel="apple-touch-icon" sizes="76x76" href="/apple-icon-76x76.png">
    <link rel="apple-touch-icon" sizes="114x114" href="/apple-icon-114x114.png">
    <link rel="apple-touch-icon" sizes="120x120" href="/apple-icon-120x120.png">
    <link rel="apple-touch-icon" sizes="144x144" href="/apple-icon-144x144.png">
    <link rel="apple-touch-icon" sizes="152x152" href="/apple-icon-152x152.png">
    <link rel="apple-touch-icon" sizes="180x180" href="/apple-icon-180x180.png">
    <link rel="icon" type="image/png" href="/favicon-32x32.png" sizes="32x32">
    <link rel="icon" type="image/png" href="/android-chrome-192x192.png" sizes="192x192">
    <link rel="icon" type="image/png" href="/favicon-96x96.png" sizes="96x96">
    <link rel="icon" type="image/png" href="/favicon-16x16.png" sizes="16x16">
    <link rel="manifest" href="/manifest.json">
    <meta name="apple-mobile-web-app-title" content="Elastic">
    <meta name="application-name" content="Elastic">
    <meta name="msapplication-TileColor" content="#ffffff">
    <meta name="msapplication-TileImage" content="/mstile-144x144.png">
    <meta name="theme-color" content="#ffffff">
    <meta name="naver-site-verification" content="936882c1853b701b3cef3721758d80535413dbfd">
    <meta name="yandex-verification" content="d8a47e95d0972434">
    <meta name="localized" content="true">
    <meta name="st:robots" content="follow,index">
    <meta property="og:image" content="https://www.elastic.co/static/images/elastic-logo-200.png">
    <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
    <link rel="icon" href="/favicon.ico" type="image/x-icon">
    <link rel="apple-touch-icon-precomposed" sizes="64x64" href="/favicon_64x64_16bit.png">
    <link rel="apple-touch-icon-precomposed" sizes="32x32" href="/favicon_32x32.png">
    <link rel="apple-touch-icon-precomposed" sizes="16x16" href="/favicon_16x16.png">
    <!-- Give IE8 a fighting chance -->
    <!--[if lt IE 9]>
    <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
    <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->
    <link rel="stylesheet" type="text/css" href="/guide/static/styles.css">
  </head>

  <!--© 2015-2021 Elasticsearch B.V. Copying, publishing and/or distributing without written permission is strictly prohibited.-->

  <body>
    <!-- Google Tag Manager -->
    <script>dataLayer = [];</script><noscript><iframe src="//www.googletagmanager.com/ns.html?id=GTM-58RLH5" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
    <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= '//www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-58RLH5');</script>
    <!-- End Google Tag Manager -->

    <!-- Global site tag (gtag.js) - Google Analytics -->
    <script async src="https://www.googletagmanager.com/gtag/js?id=UA-12395217-16"></script>
    <script>
      window.dataLayer = window.dataLayer || [];
      function gtag(){dataLayer.push(arguments);}
      gtag('js', new Date());
      gtag('config', 'UA-12395217-16');
    </script>

    <!--BEGIN QUALTRICS WEBSITE FEEDBACK SNIPPET-->
    <script type="text/javascript">
      (function(){var g=function(e,h,f,g){
      this.get=function(a){for(var a=a+"=",c=document.cookie.split(";"),b=0,e=c.length;b<e;b++){for(var d=c[b];" "==d.charAt(0);)d=d.substring(1,d.length);if(0==d.indexOf(a))return d.substring(a.length,d.length)}return null};
      this.set=function(a,c){var b="",b=new Date;b.setTime(b.getTime()+6048E5);b="; expires="+b.toGMTString();document.cookie=a+"="+c+b+"; path=/; "};
      this.check=function(){var a=this.get(f);if(a)a=a.split(":");else if(100!=e)"v"==h&&(e=Math.random()>=e/100?0:100),a=[h,e,0],this.set(f,a.join(":"));else return!0;var c=a[1];if(100==c)return!0;switch(a[0]){case "v":return!1;case "r":return c=a[2]%Math.floor(100/c),a[2]++,this.set(f,a.join(":")),!c}return!0};
      this.go=function(){if(this.check()){var a=document.createElement("script");a.type="text/javascript";a.src=g;document.body&&document.body.appendChild(a)}};
      this.start=function(){var a=this;window.addEventListener?window.addEventListener("load",function(){a.go()},!1):window.attachEvent&&window.attachEvent("onload",function(){a.go()})}};
      try{(new g(100,"r","QSI_S_ZN_emkP0oSe9Qrn7kF","https://znemkp0ose9qrn7kf-elastic.siteintercept.qualtrics.com/WRSiteInterceptEngine/?Q_ZID=ZN_emkP0oSe9Qrn7kF")).start()}catch(i){}})();
    </script><div id="ZN_emkP0oSe9Qrn7kF"><!--DO NOT REMOVE-CONTENTS PLACED HERE--></div>
    <!--END WEBSITE FEEDBACK SNIPPET-->

    <div id="elastic-nav" style="display:none;"></div>
    <script src="https://www.elastic.co/elastic-nav.js"></script>

    <!-- Subnav -->
    <div>
      <div>
        <div class="tertiary-nav d-none d-md-block">
          <div class="container">
            <div class="p-t-b-15 d-flex justify-content-between nav-container">
              <div class="breadcrum-wrapper"><span><a href="/guide/" style="font-size: 14px; font-weight: 600; color: #000;">Docs</a></span></div>
            </div>
          </div>
        </div>
      </div>
    </div>

    <div class="main-container">
      <section id="content">
        <div class="content-wrapper">

          <section id="guide" lang="en">
            <div class="container">
              <div class="row">
                <div class="col-xs-12 col-sm-8 col-md-8 guide-section">
                  <!-- start body -->
                  <div class="page_header">
<p>
  <strong>WARNING</strong>: The 2.x versions of Elasticsearch have passed their
  <a href="https://www.elastic.co/support/eol">EOL dates</a>. If you are running
  a 2.x version, we strongly advise you to upgrade.
</p>
<p>
  This documentation is no longer maintained and may be removed. For the latest
  information, see the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">current
  Elasticsearch documentation</a>.
</p>
</div>
<div id="content">
<div class="breadcrumbs">
<span class="breadcrumb-link"><a href="index.html">Elasticsearch: The Definitive Guide [2.x]</a></span>
»
<span class="breadcrumb-link"><a href="getting-started.html">Getting Started</a></span>
»
<span class="breadcrumb-link"><a href="inside-a-shard.html">Inside a Shard</a></span>
»
<span class="breadcrumb-node">Dynamically Updatable Indices</span>
</div>
<div class="navheader">
<span class="prev">
<a href="making-text-searchable.html">« Making Text Searchable</a>
</span>
<span class="next">
<a href="near-real-time.html">Near Real-Time Search »</a>
</span>
</div>
<div class="section">
<div class="titlepage"><div><div>
<h2 class="title">
<a id="dynamic-indices"></a>Dynamically Updatable Indices<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch-definitive-guide/edit/2.x/075_Inside_a_shard/30_Dynamic_indices.asciidoc">edit</a>
</h2>
</div></div></div>
<p>The next problem that needed to be solved was how to make an inverted index
updatable without losing the benefits of immutability?  The answer turned out
to be: use more than one index.</p>
<p>Instead of rewriting the whole inverted index, add new supplementary indices
to reflect more-recent changes. Each inverted index can be queried in turn—​starting with the oldest—​and the results combined.</p>
<p>Lucene, the Java libraries on which Elasticsearch is based, introduced  the
concept of <em>per-segment search</em>.  A <em>segment</em> is an inverted index in its own
right,  but now the word <em>index</em> in Lucene came to mean a <em>collection of
segments</em> plus a <em>commit point</em>—a file that lists all known segments, as depicted in <a class="xref" href="dynamic-indices.html#img-index-segments" title="A Lucene index with a commit point and three segments">Figure 16, “A Lucene index with a commit point and three segments”</a>. New documents are first added to an in-memory indexing buffer, as shown in <a class="xref" href="dynamic-indices.html#img-memory-buffer" title="A Lucene index with new documents in the in-memory buffer, ready to commit">Figure 17, “A Lucene index with new documents in the in-memory buffer, ready to commit”</a>, before being written to an on-disk segment, as in <a class="xref" href="dynamic-indices.html#img-post-commit" title="After a commit, a new segment is added to the commit point and the buffer is cleared">Figure 18, “After a commit, a new segment is added to the commit point and the buffer is cleared”</a></p>
<div id="img-index-segments" class="imageblock">
<div class="content">
<img src="images/elas_1101.png" alt="A Lucene index with a commit point and three segments">
</div>
<div class="title">Figure 16. A Lucene index with a commit point and three segments</div>
</div>
<div class="sidebar">
<div class="titlepage"><div><div>
<p class="title"><strong>Index Versus Shard</strong></p>
</div></div></div>
<p>To add to the confusion, a <em>Lucene index</em> is what we call a <em>shard</em> in
Elasticsearch, while an <em>index</em> in Elasticsearch is a collection of shards.
When Elasticsearch searches an index, it sends the query out to a copy of
every shard (Lucene index) that belongs to the index, and then reduces the
per-shards results to a global result set, as described in
<a class="xref" href="distributed-search.html" title="Distributed Search Execution"><em>Distributed Search Execution</em></a>.</p>
</div>
<p>A per-segment search works as follows:</p>
<div class="olist orderedlist">
<ol class="orderedlist">
<li class="listitem">
New documents are collected in an in-memory indexing buffer.
See <a class="xref" href="dynamic-indices.html#img-memory-buffer" title="A Lucene index with new documents in the in-memory buffer, ready to commit">Figure 17, “A Lucene index with new documents in the in-memory buffer, ready to commit”</a>.
</li>
<li class="listitem">
<p>Every so often, the buffer is <em>commited</em>:</p>
<div class="ulist itemizedlist">
<ul class="itemizedlist">
<li class="listitem">
A new segment—​a supplementary inverted index—​is written to disk.
</li>
<li class="listitem">
A new <em>commit point</em> is written to disk, which includes the name of the new
segment.
</li>
<li class="listitem">
The disk is <em>fsync’ed</em>—all writes waiting in the filesystem cache are
flushed to disk, to ensure that they have been physically written.
</li>
</ul>
</div>
</li>
<li class="listitem">
The new segment is opened, making the documents it contains visible to search.
</li>
<li class="listitem">
The in-memory buffer is cleared, and is ready to accept new documents.
</li>
</ol>
</div>
<div id="img-memory-buffer" class="imageblock">
<div class="content">
<img src="images/elas_1102.png" alt="A Lucene index with new documents in the in-memory buffer, ready to commit">
</div>
<div class="title">Figure 17. A Lucene index with new documents in the in-memory buffer, ready to commit</div>
</div>
<div id="img-post-commit" class="imageblock">
<div class="content">
<img src="images/elas_1103.png" alt="After a commit, a new segment is added to the index and the buffer is cleared">
</div>
<div class="title">Figure 18. After a commit, a new segment is added to the commit point and the buffer is cleared</div>
</div>
<p>When a query is issued, all known segments are queried in turn. Term
statistics are aggregated across all segments to ensure that the relevance of
each term and each document is calculated accurately. In this way, new
documents can be added to the index relatively cheaply.</p>
<div class="section">
<div class="titlepage"><div><div>
<h3 class="title">
<a id="deletes-and-updates"></a>Deletes and Updates<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch-definitive-guide/edit/2.x/075_Inside_a_shard/30_Dynamic_indices.asciidoc">edit</a>
</h3>
</div></div></div>
<p>Segments are immutable, so documents cannot be removed from older segments,
nor can older segments be updated to reflect a newer version of a document.
Instead, every commit point includes a <code class="literal">.del</code> file that lists which documents
in which segments have been deleted.</p>
<p>When a document is “deleted,” it is actually just <em>marked</em> as deleted in the
<code class="literal">.del</code> file. A document that has been marked as deleted can still match a
query, but it is removed from the results list before the final query results
are returned.</p>
<p>Document updates work in a similar way: when a document is updated, the old
version of the document is marked as deleted, and the new version of the
document is indexed in a new segment. Perhaps both versions of the document
will match a query, but the older deleted version is removed before the query
results are returned.</p>
<p>In <a class="xref" href="merge-process.html" title="Segment Merging">Segment Merging</a>, we show how deleted documents are purged from
the filesystem.</p>
</div>

</div>
<div class="navfooter">
<span class="prev">
<a href="making-text-searchable.html">« Making Text Searchable</a>
</span>
<span class="next">
<a href="near-real-time.html">Near Real-Time Search »</a>
</span>
</div>
</div>

                  <!-- end body -->
                </div>
                <div class="col-xs-12 col-sm-4 col-md-4" id="right_col">
                  <div id="rtpcontainer" style="display: block;">
                    <div class="mktg-promo">
                      <h3>Most Popular</h3>
                      <ul class="icons">
                        <li class="icon-elasticsearch-white"><a href="https://www.elastic.co/webinars/getting-started-elasticsearch?baymax=default&amp;elektra=docs&amp;storm=top-video">Get Started with Elasticsearch: Video</a></li>
                        <li class="icon-kibana-white"><a href="https://www.elastic.co/webinars/getting-started-kibana?baymax=default&amp;elektra=docs&amp;storm=top-video">Intro to Kibana: Video</a></li>
                        <li class="icon-logstash-white"><a href="https://www.elastic.co/webinars/introduction-elk-stack?baymax=default&amp;elektra=docs&amp;storm=top-video">ELK for Logs &amp; Metrics: Video</a></li>
                      </ul>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </section>

        </div>


<div id="elastic-footer"></div>
<script src="https://www.elastic.co/elastic-footer.js"></script>
<!-- Footer Section end-->

      </section>
    </div>

<script src="/guide/static/jquery.js"></script>
<script type="text/javascript" src="/guide/static/docs.js"></script>
<script type="text/javascript">
  window.initial_state = {}</script>
  </body>
</html>
